Efficient mems-based eye tracking system with a silicon photomultiplier sensor

ABSTRACT

Eye tracking system incorporates and/or uses one or more silicon photomultiplier (SiPM) sensor and an infrared module of a microelectromechanical (MEMs)-based scanner. The infrared module emits a beam of photons, where at least some of the photons are directed towards a user&#39;s eye while the eye tracking system is being used. The SiPM sensor(s) capture a reflection that emanates off of the user&#39;s eye.

RELATED APPLICATIONS

This application is related to co-pending U.S. patent application Ser.No. ______, filed on Feb. 9, 2018, entitled “EYE TRACKING SYSTEM FOR USEIN A VISIBLE LIGHT DISPLAY DEVICE,” and which application is expresslyincorporated herein by reference in its entirety.

BACKGROUND

Mixed-reality systems, including virtual-reality and augmented-realitysystems, have recently received significant interest for their abilityto create immersive experiences for users. Conventionalaugmented-reality (AR) systems create an augmented reality scenario bygenerating holograms that are rendered in the user's line of sight toobjects in the real world. In contrast, conventional virtual-reality(VR) systems create a more immersive experience because a user's entireview is obstructed by a virtual world.

As used herein, AR and VR systems are described and referencedinterchangeably using the umbrella term “mixed-reality system(s).”Unless specifically stated or unless specifically required, asunderstood by one of skill in the art, the descriptions herein applyequally to any and all types of mixed-reality systems, including ARsystems, VR systems, and/or any other similar system capable ofdisplaying virtual objects to a user. Accordingly, from this pointforward, the disclosure will use the term mixed-reality system todescribe any of the systems referenced above.

Of note, many mixed-reality systems use one or more on-body devices,such as a head-mounted display (hereinafter “HMD”), to render a virtualenvironment for a user. Continued advances in hardware capabilities andrendering technologies have greatly increased the realism of virtualobjects displayed within a mixed-reality environments, particularly withthe use of HMDs. For example, as the user moves their head during amixed-reality session, the rendered mixed-reality environment isautomatically updated so that the user is provided with a properperspective and view of the virtual objects in the mixed-realityenvironment.

Recent advances in this technology space relate to the use of eyetracking systems to track a movement of the user's eyes. As a result, amixed-reality system can respond not only to a user's bodily movements,but it can also respond to a user's eye movements.

However, these new eye tracking technologies are available, they areseriously lacking. In particular, the current technology is quite costlybecause it often requires additional hardware (e.g., specializedcameras) on the HMD to capture the user's eye movements. Additionally,these cameras are placed in close proximity to the eyes and typicallyobstruct the user's field of view. Furthermore, the current technologyis deficient because it consumes a large amount of battery resources. Asa result, there is a significant need to improve the eye trackingtechnology used in HMDs.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

The disclosed embodiments include eye-tracking systems and methods, suchas, but not limited to low resolution and low powermicroelectromechanical (MEMs)-based eye tracking systems and methods ofuse, which incorporate silicon photomultiplier (SiPM) sensors. ThisMEMs-based eye tracker may be optionally integrated with visible lightMEMS-based display systems. Furthermore, in some implementations, a HMDeye tracking device is integrated into the display and provides noadditional visual obstruction to the user.

Some disclosed embodiments include eye tracking systems that includes aninfrared module and one or more SiPMs. Initially, the infrared moduleemits laser light (e.g., a light wave consisting of a beam of photons).At least part of this laser light (e.g., at least some photons) isdirected towards the user's eye while the eye tracking system is beingused. After the laser light is directed towards the user's eye, then theSiPMs capture a resulting reflection. To clarify, the reflection occursas a result of the laser light initially striking and then reflectingoff of the user's eye. As the laser light is rastered across the eye,the reflected signal from each laser position is received by the SiPMand can be used to generate a greyscale image of the eye. Because theSiPMs are positioned relative to the infrared module and to the user'seye, they are able to adequately capture this reflected signal. Oncethis reflection is captured, then the user's eye position is determined.

In some embodiments, an eye tracking system is used to perform aninitial scanning of infrared light, such as with a MEMs mirrors system.This infrared light is then directed towards an eye of a user who isusing the eye tracking system. Subsequently, a reflection of theinfrared light is captured using one or more SiPMs. Of note, thisreflection is generated as a result of the infrared light being directedtowards and reflected off of the user's eye. Thereafter, an electricalresponse of the SiPMs is measured, and an image of the user's eye isgenerated using the electrical response. As multiple eye images aregenerated, which indicate a position of the eye at a specific point intime, it is possible to track the user's relative eye movements bycomparing the eye images with each other to determine a deltadisplacement of the eye between each of the multiple images.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the embodiments may be realized and obtained by means ofthe instruments and combinations particularly pointed out in theappended claims. Features of the present embodiments will become morefully apparent from the following description and appended claims, ormay be learned by the practice of the embodiments as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof various embodiments will be rendered by reference to the appendeddrawings. Understanding that these drawings depict only sampleembodiments and are not therefore to be considered to be limiting of thescope of the invention, the embodiments will be described and explainedwith additional specificity and detail through the use of theaccompanying drawings in which:

FIG. 1 shows an example architecture for providing a low resolution andlower power microelectromechanical (MEMS)-based eye tracking system thatincludes a silicon photomultiplier (SiPM) sensor.

FIGS. 2A and 2B demonstrate the differences between a specularreflection and a scatter reflection.

FIG. 3 illustrates an example digitizer module that uses a SiPM sensorto capture reflections that emanate off of a user's eye.

FIG. 4 presents a graphical readout that shows a photo-currentcorrelation between a specular reflection and a scatter reflection ascaptured by a SiPM and a specular reflection and a scatter reflection ascaptured by a PIN junction photomultiplier detector.

FIG. 5 illustrates another example digitizer module that uses a filterto filter out undesired ambient light and/or other types of light priorto that light reaching a SiPM sensor.

FIG. 6 illustrates another example digitizer module that uses a couplingcapacitor to preferably sharpen or increase the pulse response of a SiPMsensor.

FIG. 7 shows a graphical readout showing the impact on a SiPM sensor'squench time when a coupling capacitor is used and when one is not used.

FIG. 8 demonstrates that the eye tracking system can operate when thelaser module is functioning in a pulsed mode or a continuous mode.

FIG. 9 shows that when the laser module is operating in the pulsed mode,then the electrical response of a SiPM sensor can be sampled in asynchronized manner.

FIG. 10 shows that when the laser module is operating in the continuousmode, then the electrical response of a SiPM sensor can be sampledaccording to a determined frequency.

FIGS. 11A, 11B, and 11C show different example configurations of howSiPM sensors may be positioned on a HMD.

FIG. 12 illustrates an example computer system that may be used to tracka user's eye movements.

FIG. 13 provides an example method for tracking a user's eye movements.

FIG. 14 provides an example method for conserving power through the useof an IR eye tracking system.

FIG. 15 provides additional examples of techniques for performing theconservation of power described in the example method of FIG. 14.

FIG. 16 illustrates an example of a FOV display in which a portion ofthe FOV is rendered with lower resolution and a portion of the FOV isrendered with higher resolution.

DETAILED DESCRIPTION

At least some of the embodiments described herein incorporate a lowresolution and low power microelectromechanical (MEMs)-based eyetracking system that uses a silicon photomultiplier (SiPM) sensor.

The disclosed embodiments can be used to perform eye tracking. This eyetracking can be performed for the user's left eye, the user's right eye,or a combination of the user's left and right eyes. Therefore, theembodiments are not limited to tracking only a single eye. For brevity,however, the disclosure will (from this point forward) present examplesrelated to only a single eye. These examples are for illustrativepurposes only, and it will be appreciated that the principles mayequally be applied to scenarios involving more than one eye.

The disclosed embodiments may be implemented to overcome many of thetechnical difficulties and computational expenses associated withtracking a user's eye. As one example, the disclosed embodiments greatlyimprove the eye tracking technology because fewer hardware resources arerequired. To illustrate, the conventional technology often requiresadditional and/or specialized eye tracking cameras. As a result, theconventional eye tracking technology increases the amount of hardware onthe HMD. Such hardware consumes more battery resources and places moreweight on the user's head. Additionally, this hardware often obscures aportion of the user's field of view. In contrast, the disclosedembodiments significantly reduce the battery expenditure, the productioncosts, the weight factor because less hardware is used, and can beintegrated with the visible light display system of scanning MEMSsystems. For these reasons, the disclosed embodiments actually improvethe functionalities and operations of a computer system.

The disclosed embodiments may be used to perform iris authenticationbased on patterns detected from or generated from the disclosed eyetracking systems as well as generalized eye tracking for understandingthe user's gaze, attention, or intent.

Having just described various high-level attributes and advantages ofsome of the disclosed embodiments, the disclosure will now focus on FIG.1 which presents an example architecture that may be used to practicethe principles disclosed herein. Following that subject matter, thedisclosure will focus on FIGS. 2 through 11C. In particular, thesefigures illustrate various architectures and supporting illustrationsthat demonstrate multiple embodiments that provide low resolution andlow power MEMs-based eye tracking using a silicon photomultiplier (SiPM)sensor. After those figures, the disclosure will turn to FIG. 12 whichintroduces an example computer system that may be used to practice thedisclosed principles. Finally, FIG. 13 will be presented, which figureillustrates an example method for performing eye tracking.

Example Architecture(s)

FIG. 1 shows an eye tracker system 100 that includes multiple lasers105. To illustrate, the lasers 105 include, but are not limited to, aninfrared (IR) laser and lasers 105A, 105B, and 105C. These lasers 105are configured to emit “laser light.” By “laser light,” it is meant thatthe lasers 105 are able to emit a light wave having a determinedwavelength and consisting of a beam of photons. Therefore, as usedherein, the terms “laser light,” “light wave,” and “beam of photons” areinterchangeable terms.

Turning first to the IR laser, the IR laser may be a fast modulatinglaser diode that emits an infrared light wave. An infrared light wavehas a wavelength that extends from 700 nanometers (nm) up to 1000 nm. Incontrast, the lasers 105A, 105B, and 105C are lasers that emit lightwaves having other wavelengths.

As an example, the laser 105A may be a laser diode that emits a redlight wave having a wavelength extending between the range of about 630nm up to about 700 nm. The laser 105B may be a laser diode that emits agreen light wave having a wavelength extending between the range ofabout 500 nm up to about 535 nm. Finally, the laser 105C may be a laserdiode that emits a blue light wave having a wavelength extending betweenthe range of about 430 nm up to about 465 nm. Therefore, the eye trackersystem 100 may include RGB (Red, Green, Blue) laser diodes as describedabove. Other spectrums of light may also be used for the RGB lasers,utilizing different ranges than those specified above.

Although FIG. 1 shows the lasers 105 as separate diode units, theembodiments are not limited solely to such a configuration. Forinstance, the lasers 105A, 105B, and 105C may be embodied as anintegrated diode unit as opposed to three separate diode units. Thisintegrated diode unit may also operate in combination with a separate IRlaser.

Alternatively, all of the lasers 105 (i.e. the IR laser, laser 105A,laser 105B, and laser 105C) may be embodied within a single integrateddiode unit, with or without the IR laser. Here, this single integrateddiode unit is able to dynamically adjust its wavelength setting tothereby change the wavelength of the light wave that it is emitting.Accordingly, from this portion of the disclosure, it is evident that thelasers 105 may be embodied in various different forms.

To continue, FIG. 1 also shows that the lasers 105 are each emittinglaser light. In the scenario presented in FIG. 1, each of the laserlights initially passes through a collimating optic (e.g., thecollimators 110). By way of introduction, a collimator is a type of lensthat reduces the divergence angle of the highly divergent light emittedby the laser diodes. When this occurs, the rays of the light wave becomemore parallel and/or aligned with each other.

FIG. 1 shows that the eye tracker system 100 includes a collimator foreach emitted light wave. As a result, the collimators 110 narrow each ofthe emitted light waves. In situations where there is only a singleemitted light wave, only a single collimator will be used. Accordingly,in some embodiments, the number of collimators 110 may correspond to thenumber of emitted light waves.

Alternatively, a single collimator may be used to narrow multiple lightwaves at the same time. To clarify, in some embodiments, the fourcollimators 110 shown in FIG. 1 may actually be replaced by a singlecollimator. Here, this single collimator will be structured to receiveand narrow multiple light waves simultaneously. Using FIG. 1 as anexample, a single collimator may be used to simultaneously narrow thefour light waves that are emitted by the IR laser, the laser 105A, thelaser 105B, and the laser 105C.

The light wave being emitted from the IR laser is labeled as infraredlight wave 115. Similarly, the light waves being emitted from the lasers105A, 105B, and 105C are labeled as Red (R), Green (G), and Blue (B)respectively. As shown, the infrared light wave 115 is presented in adark bold format to emphasize its particular relevance with regard tothe examples that are discussed throughout the remaining portion of thisdisclosure. By way of a brief introduction, the remaining examples focuson the use of the infrared light wave 115 to track the user's eye.Although the remaining examples focus on the use of an infrared lightwave to track a user's eye, the embodiments are not strictly limited tousing only an infrared light wave.

To clarify, any of the other light waves may also be used to track theuser's eye. For instance, the red laser light, the green laser light,the blue laser, or various combinations of the infrared laser light, thered laser light, the green laser light, and/or the blue laser light mayalso be used to track the user's eye. For brevity, however, theremaining portion of this disclosure focuses on the use of the infraredlight wave 115 to track the user's eye. To reiterate once more, theembodiments are able to track the user's eye using a light wave havingany wavelength. They are not limited simply to using of an infraredlight wave.

Returning to FIG. 1, the eye tracker system 100 is integrated with amicroelectromechanical (MEMs)-based scanner 120. Although the lasers 105are shown as individual components, it will be appreciated that thelasers 105 may also be considered to be a part of the MEMs-based scanner120. Accordingly, the eye tracker system 100 is able to utilize anoutput of the MEMs-based scanner 120 in order to track the user's eye.In this manner, the eye tracking functionality is integrated with theHMD's display functionality (e.g., the MEMs-based scanner 120).Accordingly, the eye tracker system 100 is able to use many existinghardware components and thus reduce the amount hardware used to track auser's eye.

By way of introduction, the MEMs-based scanner 120 (i.e., a MEMs mirrorssystem) is used to scan the rendered pixels of an application using theRGB light that is emitted from the lasers 105A, 105B, and 105C. Thislight is scanned from those lasers across a region of the user's eye.Through this scanning operation, the MEMs-based scanner 120 is able torender an image that is viewable to the user. As shown in FIG. 1, theembodiments are also able to receive RGB light concurrently withinfrared light (which may be subsampled as described in more detaillater) and then scan the RGB light to render one or more display frames.

To that end, the MEMs-based scanner 120 may include a set of oscillatingmirrors. One or more mirrors in the set can harmonically oscillate in afirst direction in order to rapidly scan light in that first direction.While those mirrors are oscillating in the first direction, one or moreother mirrors can scan more slowly in a second direction that isorthogonal to the first direction. Other embodiments of the MEMs-basedscanner 120 may include only a single mirror that scans the image to theuser's eye. Regardless of how it is implemented, the MEMs-based scanner120 utilizes various optics to scan the RGB light emitted from the RGBlasers so that a rendered image is viewable for the user.

At this point, it is worthwhile to note that the system displayrequirements/settings for the scanned image generated by the MEMs-basedscanner 120 are very different from the requirements/settings of the eyetracker system 100, as shown by the content included below in Table 1.Initially, it is noted that a digitizer module (to be discussed later)is used to generate an image of a user's eye. By generating multiple eyeimages across a time period, then the disclosed embodiments are able todetect how the eye moves. In this manner, each image corresponds to aposition of the user's eye at a specific point in time. Further, theembodiments are able to use the MEMs-based scanner 120 to modify itsscan so as to render one or more display frames in accordance with theeye's current position. In this manner, these display frames may (1)include and/or respond to the eye's position and (2) include a displayresolution that is relative to the user's eye position (e.g., to performfoveated rendering). To clarify, in some instances, scanning the RGBlight to a target display includes foveated scanning/rendering.

With that understanding, it is noted that the resolution of the eyeimages used for eye tracking can be significantly smaller (e.g., 16times smaller) than the resolution of the display images used for imagerendering by the MEMs-based scanner 120. Thus, the effective fill factorfor the eye tracking image is but a fraction of the display's fillfactor (e.g., Table 1 shows the effective fill factor of the eye trackersystem 100 is only 6% as compared to 100% for the display settings).Because of this resolution disparity, the embodiments are configured togenerate a “subsampled” light wave (e.g., subsampled infrared light).

TABLE 1 Example Example Eye Units Display Settings Tracking SettingHorizontal Resolution Pix 1920 320 Vertical Resolution Pix 1280 240Operating Wavelength nm 450, 520, 639 850, 905, or 940 Frame RateHz >90 >90 Equivalent Pixel Duration % 5.5 37.5 Effective Fill Factor of% 100 ~6 The Display Estimated Illumination % NA >85% Power Savings ByOper- ating In Pulsed Mode

To clarify, because of the substantially reduced eye tracking resolutionrequirements, the embodiments are able to cause one or more of thelasers 105 to completely turn off during unused horizontal scan linesand/or to pulse laser output only when actively imaging a pixel.Therefore, the process of generating subsampled infrared light includesturning off the IR laser during unused horizontal scan lines and/orpulsing the IR laser's output (i.e. the infrared light wave 115).

As indicated by Table 1, the disclosed embodiments are able to provide alow power and low-resolution eye tracking system (e.g., the embodimentscan achieve at least 85% power savings and operate using significantlylower resolution eye images). Accordingly, the resolution of the imagesused for tracking a user's eye need not be the same as the resolution ofthe scanned content.

With that understanding, some of the disclosed embodiments generatesubsampled light waves (e.g., subsampled infrared light) and use thesesubsampled light waves to generate lower resolution images of the user'seye, which images are used to track the user's eye. Because theembodiments operate using a lower resolution eye image, the embodimentssignificantly reduce the amount of consumed power.

As shown in FIG. 1, the eye tracking system 100 may also include awaveguide 125. A waveguide is a device that confines a light wave'spropagation so that the light wave transmits only in a certaindirection. Waveguides are useful because even though they restrict alight wave so that it travels only in a certain direction, the lightwave does not lose significant image quality because of how thewaveguide is structured. To perform this action, a waveguide may usediffractive optical elements to couple light into the waveguide, totalinternal reflection to transmit the signal light to the display portionof the waveguide, and a diffractive optical element to outcouple thelight towards the user's eyes.

Accordingly, in some embodiments, the laser light from the lasers 105(i.e. the RGB light and/or the infrared light) is delivered from theMEMs-based scanner 120 to an object/eye 130 via this waveguide 125. Inparticular, FIG. 1 shows that the MEMs-based scanner 120 delivers theinfrared light wave 115 (which may be a subsampled light wave) to thewaveguide 125 at an incident angle θ_(i). Notably, because theMEMs-based scanner 120 includes mirrors that oscillate in variousdifferent directions, θ_(i) will not be a constant angle. Instead, thisangle will change in order to properly scan an image onto the user'seye.

Additionally, the waveguide 125 may also be used to project both the RGBlight and the subsampled infrared light onto the user's eye, as shown inFIG. 1. To clarify, in the scenario presented in FIG. 1, the infraredlight wave 115 is a subsampled light wave that is being scanned by theMEMs-based scanner 120 simultaneously with the RGB light. This RGB lightand/or the subsampled infrared light wave 115 is then directed to theuser's eye (e.g., the object/eye 130) via the waveguide 125.

Alternatively to using the waveguide 125, some embodiments use apartially transparent mirror that is positioned in front of the user'seye to direct the scanned light onto the eye. Regardless of whichimplementation is used, the scanned light may be directed to the user'seye without placing a scanning system immediately in front of the user'seye (which would result in obstructing the user's view).

Here, it will be appreciated that the object/eye 130 may be any object,and it is not limited solely to an eye. In the context of the eyetracker system 100, the object/eye 130 is the user's eye(s). However, inother contexts, the object/eye 130 can be a reflective object other thanan eye.

Once the subsampled infrared light wave 115 strikes the object/eye 130,then specular and diffuse reflections (as used herein, “scatter” and“diffuse” are interchangeable terms) will be generated. For example, aspecular reflection (labeled as “Specular” in FIG. 1) and one or morescatter reflections (labeled as “Scatter A,” “Scatter B,” and “ScatterC”) will be generated. A specular reflection corresponds to the “glint”of the user's eye, while the scatter reflections correspond to theuser's iris information. As a result, these reflections can also be usedto authenticate the user's iris.

Turning briefly to FIGS. 2A and 2B, these figures demonstrate thedifferences between a specular reflection and a scatter (aka “diffuse”reflection). As shown in FIG. 2A, an incident ray of light 205 maystrike a surface (e.g., an eye) at an incident angle θ_(i). When aspecular reflection occurs, then the reflected ray of light 210 willhave an angle of reflection θ_(r) that is the same as θ_(i). In otherwords, θ_(i)=θ_(r) for a specular reflection.

In contrast, FIG. 2B shows a scatter reflection scenario. Here, theincident ray of light 215 is striking a surface at an incident angle(not labeled). Due to the properties of the surface, however, multiplescattered reflections may be generated, where each scattered reflectionhas a reflection angle that is different than the incident angle. Inother words, θ_(i)≠θ_(r) for each of the scattered reflections. One ofthe multiple possible scattered reflections is labeled as scatterreflection 220.

Typically, the intensity of a specular reflection will be higher thanthe intensity of any of the scatter reflections. This aspect will bediscussed in more detail later. Furthermore, it will be appreciated thata combination of both a specular reflection and multiple scatterreflections may occur simultaneously.

Returning to FIG. 1, the subsampled infrared light wave 115 isreflected, after striking the object/eye 130, in such a manner so as tocreate a specular reflection and multiple scatter reflections. Asdiscussed earlier, the reflection angle θ_(r) for the specularreflection is equal to the incident angle θ_(i) of the infrared lightwave 115. Notably, the reflection angle θ_(r) also corresponds to theincident angle θ_(i) that the infrared light wave 115 left theMEMs-based scanner 120. This reflection angle will also change inaccordance with the oscillations of the mirrors in the MEMs-basedscanner 120.

FIG. 1 also shows that the intensity of the specular reflection ishigher than the intensity of the diffuse/scattered reflected infraredlight waves (i.e. the Specular line is bolder than the Scatter A,Scatter B, and Scatter C lines) as a result of the reflecteddiffuse/scattered light waves (i.e. Scatter A, Scatter B, and Scatter C)spreading over a larger angular subtense. This is shown by the line boldweights. Although FIG. 1 shows only three scatter reflections, it willbe appreciated that any number of diffuse/scatter reflections may begenerated. Again, it is worthwhile to mention that due to the potentialfor low optical to optical efficiency of some waveguide displays, and asa result of the diffuse reflected light being scattered over a largeangular subtense, the overall signal power and total number of photonspresented to the SiPM detector may be very small.

Once the specular and the scatter reflections are generated, then atleast some of these reflections will be captured by a digitizer module135. This digitizer module 135 may be configured in various differentways, as described later. However, regardless of how it is implemented,the digitizer module 135 is structured to capture some of thereflections that emanate off of the user's eye as a result of theinfrared light wave 115 (which may be subsampled) being directed ontothe user's eye. Some embodiments of the digitizer module 135 capture thereflected light (e.g., the photons) through the use of one or moresilicon photomultiplier (SiPM) sensors.

By way of introduction, a SiPM sensor is a type of photodiode sensorthat generates an electrical response as a result of detecting light(e.g., a photon). This electrical response can be used to measure andcharacterize the detected light. More detail on SiPMs will be presentedbelow.

The digitizer module 135 is used to capture the specular and scatterreflections from the user's eye and to generate an electrical response.This electrical response is converted into a digital signal. Additionalprocessing is performed on the digital signal in order to generate animage of the user's eye which includes position information for thateye. Therefore, as multiple eye images are generated, the user's eyeposition and movements are detected, by measuring the deltadisplacements of the eye across the multiple images.

Attention is now directed to FIG. 3, which illustrates one exampleimplementation of the digitizer module 135 of FIG. 1. As shown, thedigitizer module 300 is used to capture reflected light waves 305. Thesereflected light waves 305 are examples of the specular and scatterreflections described in relation to FIG. 1. For instance, the reflectedlight waves 305 may include a specular reflection, one or more scatterreflections, or a combination of a specular reflection and one or morescatter reflections. Of note, these reflected light waves 305 weregenerated as a result of shining a ray of light (e.g., an infrared beam)onto the user's eye, as described earlier.

In the embodiment shown in FIG. 3, these reflected light waves 305 arecaptured using one or more silicon photomultiplier (SiPM) sensors, suchas SiPM sensor 310. A SiPM sensor (e.g., SiPM sensor 310) is anelectronic device that converts light to electricity. Specifically, aSiPM sensor is a solid-state device that is able to detect photons on anindividual level. It is a photosensitive PN junction built on a siliconsubstrate and uses multiple microcells in the form of avalanchephotodiodes that are electrically connected together in parallel.Because of the avalanche photodiodes, the SiPM is able to operate in an“avalanche mode” (and more specifically a “Geiger” mode (more detail tofollow)) when capturing light (e.g., the reflections reflecting from theuser's eye).

A SiPM sensor is an analog device because the output of each of themicrocells is read in a parallel manner even though the device isstructured to operate in a digital switching mode. SiPM sensors areparticularly beneficial because they provide a high gain signal with arelatively low voltage output. Additionally, they provide a very fastresponse. To clarify, a SiPM sensor has a fast response regardless of asignal's intensity due to its rapid avalanche process and quenching(discussed in more detail below) of the individual microcells. Thisallows a SiPM sensor to run with a much higher modulation frequency anda much higher output signal than standard large area photodetectors.Additionally, because the SiPM sensor includes multiple detectormicrocells that fire/operate in parallel, the SiPM sensor acts as ananalog device, and the total resulting photo-current is equivalent tosampling a signal (e.g., a continuous signal) at a determined frequencywhich is how an analog device operates. Therefore, in this manner, theSiPM sensor operates as an analog device.

Because a SiPM sensor has a high gain, the output signal of the SiPMsensor can be loaded onto a flex circuit right away as opposed to havingto first pass through an additional amplifier (e.g., a trans-impedanceamplifier). Because the embodiments do not require a trans-impedanceamplifier to be placed right next to the SiPM sensor (though atrans-impedance amplifier may still be used, if desired), theembodiments simplify the design process and make the eye tracking systemconsume less power compared to the traditional approaches. As the IRlaser light is transmitted through the waveguide display, the overallassembly is less noticeable to a user.

As indicated above, a photomultiplier (e.g., a SiPM) is able to operatein an “avalanche mode.” Notably, an avalanche mode actually includes twodifferent modes, one mode occurs below breakdown and the other modeoccurs above breakdown. Breakdown refers to the point at which aphotomultiplier's gain progresses toward infinity. In most applications,infinite gain is not actually achievable. As a result, a threshold value(often a voltage value, or a “voltage breakdown”) is defined toestablish when breakpoint occurs.

The mode that occurs above the breakdown is referred to as the “Geigermode,” which is the mode that the SiPM sensor typically operates in. ASiPM sensor is able to operate in the Geiger mode because it isexternally biased. As discussed earlier, a SiPM sensor includes manymicrocells that operate in parallel. Each microcell is a combination ofa series of avalanche photodiodes and a quenching resistor. Becausethese microcells are connected in a parallel manner, the SiPM sensorincludes both a cathode (e.g., the cathode 315 shown in FIG. 3) and ananode (e.g., the anode 320). Because of the external bias, the avalanchephotodiodes operate above the breakdown which causes the SiPM sensor tooperate in the Geiger mode. Therefore, as a result of operating in theGeiger mode, a SiPM sensor provides a relatively high gain.Additionally, the correlation between the gain and the breakdown isgenerally linear.

Because the SiPM sensor operates in the Geiger mode, there is an opticalgain associated with the SiPM sensor's output signal (i.e. theelectrical response). This gain increases the output signal's intensity.Such an increase in the signal's intensity allows for the selection ofan analog to digital converter (hereinafter “ADC”) that uses less powerand that is less costly to fabricate. As a result, the disclosedembodiments significantly reduce how much power is required to track auser's eye. The embodiments also significantly reduce the manufacturingcosts because less complex (and therefore cheaper) ADCs can be used.

Returning to FIG. 3, this figure shows that the digitizer module 300includes a SiPM sensor 310 as described above. Additionally, thedigitizer module 300 includes a load resistor 325 and an ADC 330 thatmeasures/samples the voltage across the load resistor 325. In otherwords, the ADC 330 is used to sample the electrical response of the SiPMsensor 310. Some embodiments also include one or more analog low and/orhigh pass filter(s) for filtering the electrical response of the SiPMsensor 310 prior to sampling the electrical response with the ADC 330.The filter(s) effectively reduce(s) the signal to noise ratio andimproves the output of the ADC 330.

In this manner, the disclosed embodiments are able to convert light toan electrical response. Indeed, by measuring the specular and scatterreflections, the disclosed embodiments are able to generate theelectrical response which can then be used to generate an image of theuser's eye. This image captures the position of the user's eye at aspecific point in time. As multiple images are generated, then theembodiments are able to track the user's eye movements by determiningthe delta displacement of the eye across the multiple images.

As illustrated in the Figures, some embodiments include eye trackingsystems that include an infrared module (e.g., the IR laser shown inFIG. 1) and one or more SiPM sensors (e.g., the SiPM sensor 310 shown inFIG. 3). This infrared module is able to emit a light wave (e.g., a beamof photons) which, in some instances, is a subsampled infrared lightwave.

This subsampled infrared light wave is directed toward a user's eyeduring use of the eye tracking system. For example, the subsampledinfrared light wave can be directed to the user's eye through use of thewaveguide 125 shown in FIG. 1. Additionally, the process of scanning asubsampled light wave may be performed at a MEMS-based scanner systemthat includes one or more lasers.

Furthermore, one or more SiPM sensors can be positioned relative to theinfrared module and relative to the user's eye so as to capture areflection that emanates off of the user's eye. In some instances, thereflections that are captured by the one or more SiPM sensors include aspecular reflection, one or more diffuse/scatter reflections, or acombination of both a specular reflection and one or more scatterreflections, as described earlier. The one or more SiPM sensors thencapture these reflections and measure them (e.g., by generating anelectrical response).

Although FIG. 3 shows a scenario in which a single SiPM sensor, a singleload resistor, and a single ADC are being used, the embodimentsdisclosed herein are not so limited. For instance, the embodiments areable to support multiple SiPM sensors, multiple load resistors, andmultiple ADCs. Each ADC corresponds to one of the SiPM sensors. In thismanner, a single ADC is able to sample the electrical response from asingle SiPM sensor. Furthermore, this sampling can occur at a determinedfrequency. In some instances, the eye tracking system also includes ananalog low pass filter that filters the SiPM sensor's electricalresponse before the ADC samples that response. Such a configurationadvantageously reduces the signal to noise ratio and improves the outputof the ADC.

In situations where there are multiple SiPM sensors and multiple ADCs,each ADC may sample the electrical response of its corresponding SiPMsensor at a unique/different frequency. Therefore, in some embodiments,each of the multiple ADCs samples an electrical response at a frequencythat is different than the other ADCs. Alternatively, the ADCs could allsample their respective SiPM sensors at the same frequency. Accordingly,the depiction shown in FIG. 3 is for illustrative purposes only andshould not be considered as limiting the scope of the claims.

Some embodiments alternatively or additionally include a PIN junctiondetector (hereinafter a “PIN PD”). A PIN PD is another type of lightdetection device. Typically, PIN PDs do not have the gain of a SiPMsensor, and detected photons are converted into electrons. For example,as the light intensity through the waveguide is measured in the uWlevel, and the reflected light intensity from the eye and measured bythe PIN PD is in the 10-100 nW range (as the PIN PDs are removed fromthe eye, and the area of the PIN PD is relatively small), thecorresponding output current level from the PIN PD is in the 10-100 nArange, resulting in a nA scale photo-current for scattering reflectionsand a μA scale for the specular (i.e. the glint) reflection. When animpedance load of 50 ohms is used (e.g., to avoid large RC circuitdelays), then the PIN PD's smaller photo-current might have to beamplified 1000× for scatter reflections and 10× for the specularreflection. This amplification is performed to ensure that the resultingvoltage (of the PIN PD) is larger than 1 mV so that it can be properlyloaded onto a flex circuit with a typical noise profile and so thatsampling by the ADC can be performed without suffering a largequantization error. Although PIN PDs have a lower gain, they are rathereasy to fabricate and can be done so in a relatively cheap manner.

As indicated earlier, the intensity of a scatter reflection is lowerthan the intensity of a specular reflection. As a result, the measuredphoto-current of a scatter reflection will be lower than the measuredphoto-current of the specular reflection. FIG. 4 provides an exampleillustration of this occurrence.

In particular, FIG. 4 shows a graphical representation of the measuredphoto-current of both a specular measurement and a scatter measurementas captured by a SiPM sensor. As indicated, the scatter measurement islower than the specular measurement. Additionally, FIG. 4 also shows thephoto-current of a specular measurement and a scatter measurement when aPIN PD is used. Here, this figure illustrates the gain differencesbetween a SiPM sensor and a PIN PD. As shown, the gain is significantlyhigher for the SiPM sensor than it is for the PIN PD.

Due to the relatively low photo-current of the PIN PD, additionaloperational amps (“op amps”) may be used to achieve the high gainbandwidth (e.g., in the 18 GHz range) to provide the roughly 60× gain inthe 70 MHz range. Using an op amp has tradeoffs, however, because thequiescent power for an op amp is typically in the ˜100 mW range.Further, two of these op amps are cascaded to get the 3,600× gain toadequately amplify the nA current levels of the PIN PD. Because thedisclosed embodiments operate using at least some SiPM sensors, theembodiments significantly improve the battery life of the eye trackingsystem when compared to systems that use only PIN PDs, as generallyshown by the values in Table 2.

TABLE 2 Required Fre- Power Measured Gain quency Consumption TechnologyCurrent (V/V) Range Per Amplifier PIN PD 10-100 nA 1000 10 kHz- 2x OpAmps 130 MHz connected in series. Total Consumed Power = 224 mW SiPM3-30 μA 10 10 MHz- Total Consumed 130 MHz Power = 10 mW

With that understanding, some of the disclosed embodiments use acombination of SiPM sensors and PIN PDs. In these embodiments, the PINPDs may be used to operate on/measure the specular reflection, which hasa relatively higher intensity (and thus requires less gainamplification), while the SiPM sensors may be used to operate on thescatter reflections, which have relatively lower intensities. As aresult, the embodiments are able to incorporate the high gain benefitsof the SiPM sensors to operate on the scatter reflections and toincorporate the low cost/ease of production benefits of the PIN PDs tooperate on the specular reflections.

It will be appreciated, however, that other configurations are alsoavailable. For example, some of the SiPM sensors may operate on thespecular reflection and some may operate on the scatter reflections.Additionally, some of the PIN PDs may operate on the specular reflectionand/or the scatter reflections. As a result, the embodiments may causethe SiPM sensors and/or the PIN PDs to operate on different types ofreflections.

Turning now to FIG. 5, this figure illustrates another exampleimplementation of the digitizer module 135 of FIG. 1. To illustrate,FIG. 5 shows a digitizer module 500 that includes many of the samecomponents that were discussed in relation to the digitizer module 300of FIG. 3. Because many of the components are the same, the commoncomponents will not be re-labeled.

In contrast to the digitizer module 300 of FIG. 3, the digitizer module500 (which may include SiPMs and/or PIN PDs) additionally includes afilter 505. This filter can be structured to filter out light waves ofdifferent wavelengths that are different from the illuminationwavelength. As a first example, the filter 505 can be used to filter outundesired ambient light 510. While the scatter and specular infraredreflections from the illumination light source (or laser) are allowed topass through the filter 505, the undesired ambient light 510 is filteredout such that it is not able to reach the SiPM sensor. In this manner,some of the disclosed embodiments limit the amount of light waves thatreach the SiPM sensor.

In addition to filtering out the undesired ambient light 510, the filter505 may additionally or alternatively be structured to filter out otherlight waves. For example, the filter 505 may be structured to filter outred light, green light, and/or blue light from the scanning MEMs laserdisplay. Additionally or alternatively, the filter 505 may be structuredto filter out light waves that have other wavelengths in the visiblespectrum from the ambient environment. Furthermore, the filter 505 maybe structured to filter out light waves having a wavelength locatedwithin a specific portion of the infrared wavelength spectrum (or anyother spectrum). Accordingly, from this disclosure, it will beappreciated that the filter 505 may be used to filter out any type ofundesired light wave.

As indicated above, some embodiments include the filter 505 (e.g., aninfrared filter). As shown in FIG. 5, this filter 505 is disposed at alocation so that it can filter light before the light reaches the SiPMsensor.

Attention will now be directed to FIG. 6, which illustrates anotherexample implementation of a digitizer module (such as the digitizermodule 135 of FIG. 1). Here, however, the digitizer module 600 includesa first load resistor 605, a first ADC 610, a coupling capacitor 615, asecond load resistor 620 that is placed in parallel with the first loadresistor 605, and a second ADC 625. Such a configuration is particularlybeneficial when the lasers operate in a pulsed mode, which will bedescribed in more detail later.

One advantage of a SiPM sensor is that it has a very narrow pulseresponse (e.g., in the range of 15 ns). This narrow pulse response isequivalent to the “quenching time” of a firing pixel. As used herein,the “quenching time” is the time needed to recharge and restore the SiPMsensor's single photon sensitivity.

Here, if the electrical response of the SiPM sensor is coupled through asmall capacitor (e.g., the coupling capacitor 615), then an evennarrower pulse (˜1 ns) can be achieved, albeit with a similar recoverytime prior to the next pulse as with the standard load resistor. Withthis configuration, the signal on the first load resistor 605 (assampled by the first ADC 610) and the signal on the second load resistor620 (as sampled by the second ADC 625) can be easily integrated andsampled, which is beneficial when pulsed laser light is detected.

FIG. 7 shows a graphical representation of the quench times for the twoload resistors (i.e. the first load resistor 605 and the second loadresistor 620). Here, the top graph shows the quench time for the firstload resistor 605 (i.e. the response across the load resistor placedbefore the coupling capacitor). The bottom graph shows the quench timefor the second load resistor 620 (i.e. the response across the loadresistor placed after the coupling capacitor). As described above,although the pulse for a SiPM sensor is quite fast (as shown in the topfigure), the pulse will be even narrower if a coupling capacitor isused. In this manner, the time required to “reset” the SiPM sensor canbe significantly reduced.

Attention will now be focused on embodiments that are selectivelyoperable in a pulsed mode and a continuous wave mode. With reference toFIG. 1, in some instances, the disclosed embodiments are able to adjusthow the lasers 105 emit the light waves, to operate in either a pulsemode or a continuous wave mode. Accordingly, FIG. 8 shows an eyetracking system 800 that is configured to operate in either a pulsedmode 805 or a continuous wave mode 810.

When operating in the pulsed mode 805, the embodiments cause lasers(e.g., the lasers 105 shown in FIG. 1) to emit pulses of light waves asopposed to emitting a continuous wave. FIG. 9 shows a graphicalrepresentation of a digitizer module's electrical response when thelasers are configured to operate in the pulsed mode 805. In particular,FIG. 9 shows three time periods (Period 1, Period 2, and Period 3). Theellipses 905 demonstrates that the embodiments are able to operateacross any number of time periods.

During Period 1, the lasers emit light waves, and the digitizer moduleis able to detect a first amplitude peak. During Period 2, the lasers donot emit light waves. As a result, the digitizer module does not detectan amplitude peak. Finally, during Period 3, the lasers again emit lightwaves, and the digitizer module is able to detect a second amplitudepeak.

By operating a digitizer module in combination with pulsing lasers, thedigitizer module is able to generate a pulsing response that correspondsto the pulse of the lasers (e.g., see the pulsing responses shown inFIG. 9). By synchronizing the laser's clock with the sampling frequencyperformed by the digitizer module (e.g., specifically by the ADC), theembodiments are able to obtain the intensity for the reflected lightwaves at a specific position. This position corresponds to where thelight wave is being steered by a MEMs-based scanner. Because thedigitizer module is sampling at a reduced rate (e.g., only duringPeriods 1 and 3 as opposed to all of the Periods), then the embodimentsare able to preserve the system's battery life. Accordingly, someembodiments cause the lasers to operate in a pulsed mode and also causethe digitizer modules to be synchronized with the pulsing frequency ofthe lasers.

To clarify, from the above disclosure, it will be appreciated that someembodiments cause an infrared module to emit a pulsed beam of photonsand cause the detector and digitizer module (which includes one or moreSiPM sensors) to capture a pulsed beam reflection. In other words,because the incident beam is pulsed, so too will the reflected beam bepulsed. When operating in the pulsed mode, the embodiments sample theelectrical response of the SiPM sensors using one or more ADCs. Further,the sampling frequency of the one or more ADCs is synchronized with afrequency at which the pulsed light wave is generated. In this mannerthe embodiments limit the bandwidth of the ADC (and, if atrans-impedance amplifier is used, then its bandwidth is also limited)to match the modulation frequency of the lasers.

Accordingly, by (1) generating a subsampled light wave, (2) operatingthe laser in pulsed mode so that it emits light only when activelyimaging a pixel, and (3) limiting the bandwidth of the ADC (andtrans-impedance amplifier) to match the modulation frequency of thelaser, then the embodiments are able to reduce power consumption as muchas 200 mW per SiPM sensor as compared to a pulsed approach that usesstandard silicon photomultiplier detectors. Furthermore, by operating inpulsed mode with components that have a high gain, the embodimentsreduce the operational requirements on the ADC. Because SiPM sensorsprovide this high gain, the embodiments, when operating in the pulsedmode, can utilize a narrow-band ADC (and trans-impedance amplifier, ifdesired) and thus improve the system's battery life.

Some implementations cause the lasers to be pulsed only for the imageresolution that is needed for the application (e.g., perhaps 250×250pixels) instead of the full resolution/FOV (field of view) for thedisplay. Such a configuration saves power on the laser, the ADCs, andthe SiPMs (because those devices can be turned off when the laser is notemitting light, thus saving power on their end as well). Furthermore,the lasers may be pulsed only for the pixel columns that are needed asopposed to all of the pixels.

In an alternative implementation, the eye tracking system can beconfigured in a state that uses lower resolution and lower powertechniques in a large region of the image while in other parts of theimage FOV, full resolution scanning techniques are used but only for thepart of the image that is of most importance (e.g., turn the laser onand oversample the SiPM output for only those important image areas) (asgenerally shown in FIG. 16).

In yet another implementation, when the laser is turned off or powereddown during the pulsing, then the power supplies for the laser, SiPMs,and/or ADCs are also turned off or powered down. In some instances,these devices are also turned off or powered down for unneededhorizontal/vertical scanning blanking periods. By turning these devicesoff or powering them down, then additional power savings may berealized. Furthermore, the ADC readout and sampling may be synchronizedwith the laser pulses. Even further, the ADCs may be turned off (or putin a low power state) during blanking periods (e.g., vertical blankingperiods).

In an alternative embodiment and/or at an alternative time, the lasersmay be configured to emit a continuous light wave (e.g., while operatingin the continuous mode 810 illustrated in FIG. 8). As discussed earlier,a SiPM sensor operates as an analog device. As a result, the SiPM sensoris also able to operate on a continuous light wave. Such a configuration(1) improves signal levels that reduce susceptibility to electricalsignal integrity issues and (2) reduces the power consumption becausethe SiPM sensor provides a higher intensity signal as a result of itshigh gain characteristics. One example operation is shown in FIG. 10.

Here, this figure shows a digitizer module operating in a continuousmode 1000. Although each firing detector cell of a SiPM sensor has asimilar quenching time to that which was mentioned above, the SiPMsensor can nevertheless operate as an analog device because it usesmultiple firing detector cells. This results in a device that cancontinuously sample a signal at a certain frequency so as to generatethe smooth electrical response 1005 line. Of note, when the embodimentsoperate in the continuous mode 1000, the ADC sampling may beproportional to the resolution of the eye tracking camera frame.Accordingly, FIG. 10 shows an electrical response 1005 of a SiPM sensor.Here, the sampling locations are shown by the multiple dots (e.g., thedot 1010 is one sampling location). Further, the lined arrows 1015demonstrate the sampling frequency of an ADC as it samples theelectrical response 1005. For clarity, the ADC performs the sampling onthe electrical response 1005.

The embodiments may also improve the signal to noise ratio to addressthe high frequency sampling peaks when a SiPM sensor is used as ananalog device. For example, in some situations, the embodiments performdouble or multiple frequency sampling on the output signal (by an ADC)from the SiPM sensor (i.e. oversampling). When the output signal isoversampled, then the embodiments can also perform a running averagesmooth operation to avoid high frequency noise influences. Additionallyor alternatively, some embodiments (as discussed earlier) include ananalog low pass filter that filters the SiPM sensor's electricalresponse prior to being sampled by an ADC. In this manner, theembodiments provide a high-quality signal for the ADC to sample.Accordingly, the embodiments are also able to operate in a continuousmode, as described above.

Building on that understanding, using a low pass filter to filter theeye tracking image can be achieved in a variety of ways. For example,hardware filtering may be performed in which additional hardware filtersat used in conjunction with the ADC. In such a scenario, additionalcapacitance may be added to the input of the ADC to smooth the responseof the high frequency noise as discussed earlier. Additionally oralternatively, software filtering may be performed (e.g., with a cameraimage signal processing unit). To illustrate, prior to the eye positioncalculation being performed, the embodiments are able to run an “edgepreserving filter” (e.g., a joint bilateral filter) to improve thesignal to noise ratio of the system. Given that the eye image has alower image resolution, any latency should be very low and can beperformed in real-time.

From the above disclosure, it will be apparent that the disclosedembodiments are able to cause a laser to emit a continuous light wave ofphotos that is continuous for a predetermined duration (e.g., multiplemicroseconds, multiple seconds, etc.). The continuous light wave may bedirected so that it illuminates a person's eye. As a result of thisillumination, a continuous wave reflection is also generated. Then, oneor more SiPM sensors are able to capture this continuous wave reflectionduring use of the eye tracking system. As discussed earlier, theembodiments are able to scan a continuous subsampled light wave usingthe MEMs-based scanner (which includes lasers). To clarify, when theembodiments operate in continuous mode, then the continuous light wavemay also be a continuous subsampled light wave.

Attention will now be directed to FIGS. 11A through 11C. In particular,these figures illustrate various example configurations in which a SiPMsensor and a PIN PD may be positioned on a HMD.

Turning first to FIG. 11A, this figure shows a HMD 1100. In someinstances, HMD 1100 is a system that includes a target display which hasa field of view (FOV) that is visible to a user's eye during use of theHMD 1100. In some embodiments, the HMD 1100 is additionally configuredto generate images that are rendered on the target display and to alsoimage the user's eye.

FIG. 11A shows a first example implementation of an inner configurationschema for a HMD (e.g., HMD 1100A). In particular the HMD 1100A mayinclude a left eye display 1105 and a right eye display 1110. Situatedaround the left eye display 1105 and the right eye display 1110 aremultiple SiPM sensors that are represented by the circles. The trianglesrepresent PIN PDs. In this example configuration, there are a total of16 photomultipliers (8 SiPM sensors and 8 PIN PDs) that are positionedaround the eye displays in a 1:1 correlation (i.e. one SiPM sensorfollowed by one PIN PD followed by on SiPM sensor, etc.).

FIG. 11B shows another HMD 1100B that has a different configuration.Here, the pattern of SiPM sensors to PIN PDs is 2:1, meaning that twoSiPM sensors are followed by one PIN PD which is then followed by twoSiPM sensors, etc.

FIG. 11C shows another HMD 1100C that has yet another configuration.Here, the pattern of SiPM sensors to PIN PDs is 3:1, meaning that threeSiPM sensors are followed by one PIN PD which is then followed by threeSiPM sensors, etc.

Other non-limiting configurations include patterns of 4:1, 5:1, 6:1, orany other type of configuration. Furthermore, although some of the aboveconfigurations focused on situations where there was a larger number ofSiPM sensors, some embodiments may include a larger number of PIN PDs.For example, instead of 2 SiPM sensors to 1 PIN PD, some embodiments mayinclude 2 PIN PDs for every 1 SiPM sensor. Further, some embodiments mayinclude 3 PIN PDs for every 1 SiPM sensor. Accordingly, from thisdisclosure, it will be appreciated that the embodiments are able tosupport any configuration or pattern of SiPM sensors and PIN PDs.

Of note, the above embodiments focused on scenarios in which the eyedisplays were generally shaped as parallelograms. Because of this shape,the SiPM sensors and PIN PDs were also organized in a generalparallelogram shape. However, other embodiments are also conceived. Forexample, in some situations, the SiPM sensors and the PIN PDs can beorganized in an elliptical shape around the parallelogram-shaped eyedisplays. In other embodiments, the eye displays may have an ellipticalshape, and the SiPM sensors and PIN PDs may be placed around the eyedisplays in an elliptical shape with respect to one another.Accordingly, the disclosed embodiments support any type of configurationfor the SiPM sensors and the PIN PDs.

Even further, some embodiments include only SiPM sensors such that theydo not include any PIN PDs. In these embodiments, the SiPM sensors areplaced around the eye displays. Additionally, the embodiments are ableto support any number of SiPM sensors and any number of PIN PDs. As afirst example, an HMD may include a single SiPM sensor and zero, one, orany number of PIN PDs. Alternatively, an HMD may include two SiPMsensors and zero, one, two, or any number of PIN PDs. Alternatively, anHMD may include three SiPM sensors and zero, one, two, three, or anynumber of PIN PDs.

As illustrated by FIGS. 11A through 11C, the SiPM sensors and the PINPDs are positioned around the eye displays. Because of this positioning,the SiPM sensors and the PIN PDs are placed a determined distance from aplane that corresponds to the user's eye (i.e. an “eye plane”). Of note,the distance between the photomultipliers (i.e. the SiPM sensors and thePIN PDs) and the user's eye plane may be any distance. For example, thedistance may be 1 millimeter (mm), 2 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8mm, 9 mm, 10 mm, 11 mm, 12 mm, 13 mm, 14 mm, 15 mm, and so forth.

Accordingly, multiple SiPM sensors may be used in conjunction with eacheye. Such a configuration (i.e. multiple SiPM sensors) is beneficial forat least the following reasons. First, multiple SiPM sensors allow forthe detection of scatter and/or specular reflections (i.e. glints) forcorrect rendering of the eye plane. Additionally, multiple SiPM sensorsallow for an improved collection of the reflected infrared light. Thisalso improves the uniformity of the detection scheme. As a result, thepower savings that are provided by the disclosed embodiments areparticularly beneficial when multiple SiPM sensors are used.

Example Computer System

Attention will now be directed to FIG. 12, which illustrates anexemplary computing system that can incorporate and/or be used with thedisclosed embodiments. As used herein, “computer system,” “computingsystem,” and simply “computer” are similar terms that may beinterchanged with each other. Further, the computer system 1200 may takeany form. As examples only, FIG. 12 shows that the computer system 1200may take the form of a HMD 1205A, a desktop/laptop 1205B, or any othercomputing form (e.g., a stand-alone or distributed computing system).Accordingly, the ellipses 1205C demonstrates that the computer system1200 may be embodied in any form and thus it is not limited simply thatwhich is illustrated in FIG. 12.

The computer system 1200 also includes at least one hardware processingunit 1210 (aka “processor”), input/output (I/O) interfaces 1215,graphics rendering engines 1220, one or more sensors 1225 (e.g., eyetracking sensors), and storage 1230. The computer system 1200 alsoincludes various different components that are useful for tracking theuser's eye. To illustrate, the computer system 1200 includes a controlmodule 1235, a MEMS module 1240 which may include an infrared module1240A, and a digitizer module 1245. More detail on these components willbe discussed later.

The storage 1230 may be physical system memory, which may be volatile,non-volatile, or some combination of the two. As such, the storage 1230may be considered a computer-readable hardware storage device that iscapable of storing computer-executable instructions.

The term “memory” may also be used herein to refer to non-volatile massstorage such as physical storage media. If the computer system 1200 isdistributed, the processing, memory, and/or storage capability may bedistributed as well. As used herein, the term “executable module,”“executable component,” or even “component” can refer to softwareobjects, routines, or methods that may be executed on the computersystem 1200. The different components, modules, engines, and servicesdescribed herein may be implemented as objects or processors thatexecute on the computer system 1200 (e.g. as separate threads).

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such as hardware processing unit 1210)and system memory (such as storage 1230), as discussed in greater detailbelow. Embodiments also include physical and other computer-readablemedia for carrying or storing computer-executable instructions and/ordata structures. Such computer-readable media can be any available mediathat can be accessed by a general-purpose or special-purpose computersystem. Computer-readable media that store computer-executableinstructions in the form of data are physical computer storage media.Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example and not limitation, thecurrent embodiments can comprise at least two distinctly different kindsof computer-readable media: computer storage media and transmissionmedia.

Computer storage media are hardware storage devices, such as RAM, ROM,EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flashmemory, phase-change memory (PCM), or other types of memory, or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that can be used to store desired programcode means in the form of computer-executable instructions, data, ordata structures and that can be accessed by a general-purpose orspecial-purpose computer.

The computer system 1200 may also be connected (via a wired or wirelessconnection) to external sensors 1255 (e.g., one or more remote cameras,accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.).Further, the computer system 1200 may also be connected through one ormore wired or wireless networks 1250 to remote systems(s) 1260 that areconfigured to perform any of the processing described with regard tocomputer system 1200.

During use, a user of the computer system 1200 is able to perceiveinformation (e.g., a mixed-reality scene) that is scanned by the MEMsmodule 1240 or that is presented on a display included among the I/Ointerface(s) 1215. The I/O interface(s) 1215 and sensors 1225/1255 alsoinclude gesture detection devices, eye trackers, and/or other movementdetecting components (e.g., cameras, gyroscopes, accelerometers,magnetometers, acoustic sensors, global positioning systems (“GPS”),etc.) that are able to detect positioning and movement of one or morereal-world objects, such as a user's hand, a stylus, and/or any otherobject(s) that the user may interact with while being immersed in amixed-reality scene.

In some instances, the positioning and movement of the user and theobjects (both virtual and actual) are continuously monitored. Thismonitoring specifically detects any variation in the position and themovement of the objects, such as a detected change in position,velocity, orientation, or acceleration. These movements can be absolutemovements and/or relative movements, such as compared to a relativepositioning of the HMD, and such that movements/positioning of the HMDwill be calculated into the relative movements/positioning of theobjects as they are presented in the scene.

The graphics rendering engine 1220 is configured, with the hardwareprocessing unit 1210, to render one or more virtual objects within thescene. As a result, the virtual objects accurately move in response to amovement of the user and/or in response to user input as the userinteracts with the virtual objects.

A “network,” like the network 1250 shown in FIG. 12, is defined as oneor more data links and/or data switches that enable the transport ofelectronic data between computer systems, modules, and/or otherelectronic devices. When information is transferred, or provided, over anetwork (either hardwired, wireless, or a combination of hardwired andwireless) to a computer, the computer properly views the connection as atransmission medium. The computer system 1200 will include one or morecommunication channels (e.g., TCP ports, UDP ports, etc.) that are usedto communicate with the network 1250. Transmissions media include anetwork that can be used to carry data or desired program code means inthe form of computer-executable instructions or in the form of datastructures. Further, these computer-executable instructions can beaccessed by a general-purpose or special-purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

Additionally or alternatively, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-Programmable Gate Arrays(FPGAs), Program-Specific or Application-Specific Integrated Circuits(ASICs), Program-Specific Standard Products (ASSPs), System-On-A-ChipSystems (SOCs), Complex Programmable Logic Devices (CPLDs), CentralProcessing Units (CPUs), and other types of programmable hardware.

As discussed above, computer systems are able to provide a broad varietyof different functions. According to the disclosed principles, theembodiments provide further functionalities related to providing a lowresolution and low power MEMs-based eye tracking system with a siliconphotomultiplier (SiPM) sensor. Accordingly, attention will now bedirected to FIG. 13 which illustrates an example method for performingeye tracking.

Example Method(s)

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed. These methods areimplemented by one or more processors of a computer system (e.g., thecomputer system 1200 of FIG. 12). By way of example, a computer systemincludes one or more computer-readable hardware storage media that storecomputer-executable code. This computer-executable code is executable bythe one or more processors to cause the computer system to perform thesemethods

FIG. 13 illustrates an example method 1300 for generating an image of aneye of a user. Of note, method 1300 may be performed by the computersystem 1200 shown in FIG. 12.

As shown, subsampled infrared light is initially scanned (act 1305).Here, the MEMs module 1240, which includes the infrared module 1240A,may be used to generate and scan the subsampled infrared light.

Next, the subsampled infrared light is directed towards an eye of a userwho is using the eye tracking system (act 1310). Here, this act may beperformed by the control module 1235.

Subsequently, a reflection of the subsampled infrared light is capturedusing one or more silicon photomultipliers (SiPM) (act 1315). Notably,the reflection is generated as a result of the subsampled infrared lightbeing directed towards the user's eye. This act may be performed by thedigitizer module 1245.

Thereafter, an electrical response of the one or more SiPMs is measured(act 1320). This act may also be performed by the digitizer module 1245.

Finally, an image of the user's eye is generated using the electricalresponse (act 1325). In some instances, this act is performed by thecontrol module 1235 shown in FIG. 12. By generating multiple images, theembodiments are able to track the movement of the user's eye across atime period.

Other methods within the scope of this disclosure include methods forperforming iris authentication by generating an iris pattern based onthe scatter reflections from a particular user's eye and mapping thegenerated iris pattern to stored iris patterns associated withauthenticated users, which are stored locally or remotely. Then, uponfinding a match, authenticating the user or, alternatively, upon failingto find a match, refraining from authenticating the user.

Turning now to FIG. 14, an example method 1400 is illustrated forconserving power through the use of an IR eye tracking system (e.g.,perhaps the eye tracker 100 from FIG. 1 or the HMD 1100 shown in FIG.11A) that may be incorporated into a visible light display. Here, itwill be appreciated that the IR eye tracking system may be includedwithin a HMD, or it may be a system that is distinct and independentfrom a HMD. Advantages of such a process include, but are not limitedto, (1) a laser that operates at a lower resolution, (2) a laser that ispulsed, thus periodically operating in a low or no power state, and (3)a system that includes an eye tracking IR illuminator and IR detectorsthat operate in combination with high resolution/low resolution modes.Regarding this third feature, these modes may be a subset of the RGBdisplay area (as generally shown in FIG. 16). Additionally, someembodiments use the eye tracking information to adjust/track the regionof the eye image position.

As shown in FIG. 14, an RGB laser module (e.g., lasers 105A, 105B, and105C shown in FIG. 1) is used to emit RGB light (act 1405). Either inparallel with the emission of the RGB light or after the emission of theRGB light, a IR laser module (e.g., IR laser shown in FIG. 1) is used toemit IR light (act 1410). FIG. 14 shows acts 1405 and 1410 next to eachother to illustrate the fact that these two acts are not temporallydependent on each other. Furthermore, act 1410 may be performed withoutact 1405 being performed.

Next, a scanning system (e.g., MEMs-based scanner 120 from FIG. 1) isused to scan the RGB light to a target display or object (e.g., perhapsobject/eye 130 shown in FIG. 1) and to scan the IR light to a user's eye(act 1415). Following that process, an eye tracking sensor (e.g.,digitizer module 135 from FIG. 1) images the user's eye using reflectedlight that is reflected from the user's eye during the scanning of theIR light (act 1420). Next, the eye tracking system (e.g., eye tracker100 from FIG. 1) is caused to conserve its power during the scanning ofthe RGB light and/or the IR light (act 1425).

Turning now to FIG. 15, additional details are provided regarding theconservation of power described in act 1425. In particular, conservingthe power may be achieved in a variety of ways, as shown in FIG. 15. Ofnote, the processes outlined in FIG. 15 may be performed individually orin combination with any of the other described processes. As such, thereis no required dependence between these illustrated processes.

As shown, the process of conserving power (i.e. act 1425 from FIG. 14)may include any of the following. First, power may be conserved byselectively pulsing the RGB laser module (e.g., lasers 105A, 105B, and105C from FIG. 1) to render an image resolution on the target display(e.g., perhaps the object/eye 130 from FIG. 1) that is irregular and/orless than a full resolution for the entire field of view (FOV). Second,power may be conserved by selectively modifying the scanning of the RGBlight to render an image resolution on the target display that isirregular and/or less than a full resolution for the entire FOV. Third,power may be conserved by modifying the scanning of the IR light in sucha manner as to scan the IR light on the target display with an irregularresolution that is less than a full resolution for the entire FOV.Fourth, power may be conserved by selectively modifying power suppliedto the RGB module, the IR module, or the eye tracking sensor (e.g., thedigitizer module 135 from FIG. 1) during the scanning of the RGB lightor the IR light. Fifth, power may be conserved by selectively altering apower state of the RGB laser module, the IR module (e.g., the IR laserfrom FIG. 1), and/or the eye tracking sensor. In this manner, thedisclosed principles may be performed to provide a low power eyetracking system.

In addition to the method described above, a head mounted display (HMD)system (e.g., HMD 1100 from FIG. 11A) may also be provided. Here, theHMD system may include a target display having a field of view (FOV)that is visible to a user's eye during use of the HMD system. This HMDsystem may be configured to generate images that are rendered on thetarget display and to also image the user's eye. In some instances, thisHMD system may be implemented as an IR eye tracking system. In such ascenario, the IR eye tracking system may be used with a visible lightdisplay device having a target display with a FOV that is visible to auser's eye during use of the visible light display device.

This HMD system may comprise an RGB (red, green, blue) laser module(e.g., lasers 105A, 105B, and 105C from FIG. 1) that emits RGB light.The system may also include an IR (infrared) module (e.g., IR laser fromFIG. 1) that emits IR light. Additionally, the system may include ascanning system (e.g., MEMs-based scanner 120 from FIG. 1) that mayperform one or more of (a) scanning of the RGB light to the targetdisplay or (b) scanning of the IR light to the user's eye. In someinstances, scanning the IR light may be performed on a limited portionof the target display. Further, at least some of the IR light will bereflected to the user's eye.

Even further, the system may include an eye tracking sensor (e.g., thedigitizer module 135) that images the user's eye with reflected lightthat reflects from the user's eye during the scanning of the IR light.The sensor may include one or more sensors (e.g., SiPMs) that generatean electrical response corresponding to the reflected light.Additionally, the system may include a control module that conservesenergy during the scanning of either the RGB light or the IR light byperforming any of the processes outlined in FIG. 15.

In some instances, the one or more sensors described above may includeone or more SiPMs. Further, the system may also include an ADC for eachof the one or more SiPMs. Each of these ADCs may be configured to samplean electrical response for a corresponding one SiPM as described in someof the earlier figures. Additionally, the control module described abovemay be further configured to selectively modify power supplied to theone or more SiPMs and/or the ADCs during the scanning process.

In some instances, the control module conserves energy during thescanning of the RGB light by selectively pulsing the RGB laser module torender the image resolution on the target display. Additionally, someembodiments also pulse the IR module. When such pulsing occurs, thispulsing may cause either the RGB laser module or the IR module to pulseat a pulse rate that is determined to generate an image havingpre-determined image resolution.

Additional power saving benefits may be achieved by pulsing either theIR laser and/or the RGB laser for only a subset of pixels as opposed toall (or a full array) of the pixels. With regard to saving power byselectively modifying the power supplied to the eye tracking sensor,this may be achieved by either power down the sensor's power supply ortemporarily turning off the power supply.

To facilitate the generation of a lower resolution image, someembodiments cause the scanning system to scan the IR light to only oneor more selected sub-portions of the target display's FOV whilerefraining from scanning the IR light to other sub-portions of thetarget display's FOV. Such a scenario is illustrated in FIG. 16 (e.g.,foveated scanning/rendering). As such, the scanning system is able toscan different resolutions of RGB light and/or IR light to differentportions of the target display's FOV.

By practicing the principles disclosed herein, significant advantagesmay be realized. For instance, as a result of imaging the user's eye,the IR eye tracking system is able to determine an optical axis of theuser's eye (i.e. where the user is focusing his/her eyes). By imagingthe user's eye, a resolution of the target display image (which may becreated by the RGB light) may be adjusted in accordance with thedetermined optical axis. In this manner, imaging the user's eye helpsfacilitate the performance of foveated rendering.

Accordingly, the disclosed embodiments provide novel architectures andmethods for tracking a user's eye movements and for performing irisauthentication. In some embodiments, the eye tracking system includes aninfrared module (e.g., as a part of a MEMs-based scanner) and one ormore SiPM sensors. Together, these components operate to track a user'seye movements by measuring a reflection that emanates off of a user'seye. This reflection is measured using at least a SiPM sensor. As aresult, the embodiments are able to generate an image of the user's eyeusing this measurement. In this manner, the embodiments are able totrack the user's eye movements in a manner that requires a lowresolution eye image and that significantly reduces the amount ofconsumed power.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. An eye tracking system comprising: an infraredmodule that emits a beam of photons, wherein at least some of theemitted photons are directed toward a user's eye during use of the eyetracking system; and one or more silicon photomultipliers (SiPM) thatare positioned relative to the infrared module and that capture areflection of at least some photons which reflect from the user's eye asa result of the at least some photons being directed towards the user'seye within the beam of photons.
 2. The eye tracking system of claim 1,wherein the infrared module is configured to emit at least one of apulsed beam of photons or a continuous wave of photos that is continuousfor a determined duration, whereby the reflection is also either apulsed beam reflection or a continuous wave reflection, and wherein theone or more SiPMs capture the pulsed beam reflection or the continuouswave reflection during use of the eye tracking system.
 3. The eyetracking system of claim 1, wherein the eye tracking system furtherincludes a bandpass filter that filters ambient light so that theambient light is not captured by the one or more SiPMs.
 4. The eyetracking system of claim 1, wherein the eye tracking system furtherincludes an analog to digital converter (ADC) for each of the one ormore SiPMs, each ADC being configured to sample an electrical responsefor a corresponding one SiPM.
 5. The eye tracking system of claim 1,wherein the eye tracking system further includes an analog low passfilter, and wherein an electrical response of a first SiPM is filteredthrough the analog low pass filter.
 6. The eye tracking system of claim1, wherein the eye tracking system further includes a waveguide, andwherein the at least some of the emitted photons that are directedtoward the user's eye pass through the waveguide.
 7. The eye trackingsystem of claim 1, wherein the eye tracking system includes at leastthree SiPMs that are disposed in an elliptical configuration in relationto each other on a head mounted device.
 8. The eye tracking system ofclaim 1, wherein the eye tracking system further includes: a RGB (Red,Green, Blue) laser module integrated with the infrared module; and amicroelectromechanical (MEMS) scanner that is used to image a pixel foran application with RGB light emitted from the RGB module.
 9. An eyetracking system comprising: one or more processors; and one or morecomputer-readable hardware storage devices having stored thereoncomputer-executable instructions that are executable by the one or moreprocessors to cause the computer system to: scan subsampled infraredlight; direct the subsampled infrared light towards an eye of a user whois using the eye tracking system; capture, using one or more siliconphotomultipliers (SiPMs), a reflection of the subsampled infrared light,the reflection being generated as a result of directing the subsampledinfrared light towards the user's eye; measure an electrical response ofthe one or more SiPMs; and generate an image of the user's eye using theelectrical response.
 10. The eye tracking system of claim 9, wherein thereflection is at least one of a specular reflection or a scatterreflection, and wherein the subsampled infrared light is generated at amicroelectromechanical (MEMs) scanner system that includes an infraredlaser.
 11. The eye tracking system of claim 9, wherein the subsampledinfrared light is one of a pulsed light emission or a continuous wavelight emission that is continuous for a determined duration.
 12. The eyetracking system of claim 9, wherein the subsampled infrared light is apulsed light emission, and wherein execution of the computer-executableinstructions further causes the computer system to: sample theelectrical response of the one or more SiPMs using one or more analog todigital converters (ADCs), wherein a sampling frequency of the one ormore ADCs is synchronized with a frequency at which the pulsed lightemission is generated.
 13. The eye tracking system of claim 9, whereinthe subsampled infrared light is a continuous wave light emission thatis continuous for a predetermined duration, and wherein a scan angle ofthe continuous wave light emission varies.
 14. The eye tracking systemof claim 9, wherein execution of the computer-executable instructionsfurther causes the eye tracking system to: receive red green blue (RGB)light concurrently with the subsampled infrared light; and scan the RGBlight to render one or more display frames.
 15. The eye tracking systemof claim 14, wherein execution of the computer-executable instructionsfurther causes the computer system to: based on the image of the user'seye, which image corresponds to a position of the user's eye, modify thescan to render the one or more display frames to include positioncontent and resolution that is relative to the user's eye position. 16.A method for generating an image of an eye of a user, the method beingimplemented by a computer system that includes one or more processors,the method comprising: at a microelectromechanical (MEMs) system,scanning subsampled infrared light; directing the subsampled infraredlight towards the user's eye; capturing one or more reflections of thesubsampled infrared light using one or more silicon photomultipliers(SiPMs), the one or more reflections being generated as a result ofdirecting the subsampled infrared light towards the user's eye;measuring at least one electrical response of the one or more SiPMs; andgenerating at least one image of the user's eye using the at least oneelectrical response.
 17. The method of claim 16, wherein the methodfurther includes using one or more PIN junction photodetectors to alsocapture the one or more reflections.
 18. The method of claim 17, whereinthe one or more reflections include a specular reflection and at leastone scatter reflection, and wherein the one or more SiPMs capture andmeasure the at least one scatter reflection and the one or more PINjunction photomultipliers capture and measure the specular reflection.19. The method of claim 16, wherein the at least one electrical responseof the one or more SiPMs is coupled to a capacitor.
 20. The method ofclaim 16, wherein the method further includes using an analog to digitalconverter (ADC) to sample the at least one electrical response of theone or more SiPMs.